To meet the demand for wireless data traffic having increased since deployment of 4th generation (4G) communication systems, efforts have been made to develop an improved 5th generation (5G) or pre-5G communication system. Therefore, the 5G or pre-5G communication system is also called a ‘Beyond 4G Network’ or a ‘Post LTE System’.
The 5G communication system is considered to be implemented in higher frequency (mmWave) bands, e.g., 60 GHz bands, so as to accomplish higher data rates. To decrease propagation loss of the radio waves and increase the transmission distance, the beamforming, massive multiple-input multiple-output (MIMO), Full Dimensional MIMO (FD-MIMO), array antenna, an analog beam forming, large scale antenna techniques are discussed in 5G communication systems.
In addition, in 5G communication systems, development for system network improvement is under way based on advanced small cells, cloud Radio Access Networks (RANs), ultra-dense networks, device-to-device (D2D) communication, wireless backhaul, moving network, cooperative communication, Coordinated Multi-Points (CoMP), reception-end interference cancellation and the like.
In the 5G system, Hybrid FSK and QAM Modulation (FQAM) and sliding window superposition coding (SWSC) as an advanced coding modulation (ACM), and filter bank multi carrier (FBMC), non-orthogonal multiple access (NOMA), and sparse code multiple access (SCMA) as an advanced access technology have been developed.
Meanwhile, in an environment having a low Signal-to-Interference-plus-Noise Ratio (SINR), a method for modulating a signal which will be transmitted or received based on a Frequency and Quadrature-Amplitude Modulation (FQAM) is being discussed. However, since a signal modulated based on the FQAM technique occupies a wider bandwidth than a signal modulated based on a QAM technique and the like, the performance degradation due to a change in the channel of the signal based on the FQAM technique is larger than the signal modulated based on the QAM technique and the like. Therefore, a method for solving the problem is required.